1. Field of the Invention
The present invention relates to improving computational efficiency in an optimizing predictive model. In particular, the present invention relates to improving computational efficiency in an optimizing predictive model that is optimized using a stochastic gradient descent method.
2. Discussion of the Related Art
In machine learning, an optimizing predictive model is a computational model that learns a function of its input values. One type of optimizing predictive model applies a stochastic gradient descent optimization technique over a loss function. Typically, the optimization procedure involves iteratively executing the model, and then differentiating the model to adapt the values of the model parameters to minimize the loss function. The complexity of such a computation task is typically at least proportional to the size of the model. Therefore, it is desirable to have a model that is smaller, and which requires fewer computational operations.
An optimizing predictive model may be implemented, for example, in a neural network. A neural network model is usually based on a graph consisting of nodes (referred to as “neurons”) and directed, weighted edges that connect the neurons. The directed graph typically represents the function that is to be computed in the computational model. In a typical implementation, each neuron is assigned a simple computational task (e.g., a linear transformation followed by a squashing function, such as a logistic function) and the loss function is computed over the entire neural network model. The parameters of the neural network model are typically determined (“learned”) using a method that minimizes the loss function. Stochastic gradient descent is a method that is often used to achieve the minimization. In stochastic gradient descent, optimization is achieved iteratively by (a) finding analytical gradients for the loss functions and (b) perturbing or moving the test values by a small amount in the direction of the gradient, until the loss function is minimized.